feat(bundle): format v2 + groundwork for tiered storage rewrite (epic #540)#537
Merged
Conversation
Introduces .iafbt format version 2 (default writer) — see
docs/design/bundle-format-v2.md for the public spec.
Changes:
- Backtest dataclass gains an engine_type field ('vector' | 'event' |
None) plus vector_runs / event_runs / vector_metrics / event_metrics
derived properties. Engine-tagged at construction by VectorBacktest
and EventBacktest service paths; legacy / unknown bundles keep the
engine-agnostic backtest_runs / backtest_summary view.
- Bundle envelope v2 routes runs into engine-specific slots
(vector_runs / event_runs) based on engine_type. v1 envelopes
remain readable indefinitely.
- Eight heavy metric time-series (equity_curve, drawdown_series,
cumulative_return_series, rolling_sharpe_ratio, monthly_returns,
yearly_returns, twr_equity_curve, twr_drawdown_series) are
extracted into embedded Parquet blobs with int64 epoch-ms
timestamps and float64 values, replacing the v1 inline
list-of-(value, ISO-string) shape.
- save_bundle gains format_version (default 2, accepts 1 for
downgrade) and float32_ohlcv (off by default) options.
- open_bundle and Backtest.open gain summary_only=True for bulk
listing pipelines that skip the per-run blob decode.
OHLCV side store, content addressing, and LazyOhlcvDict are
unchanged. Public OHLCV dedup protocol spec added at
docs/design/ohlcv-dedup-protocol.md for upload-style integrations.
17 bundle tests pass (8 new v2-specific cases); full suite passes
1681/1681 against unchanged behaviour for v1 bundles.
Measured on real production-shape bundles (~570 KB at level 7, 7 runs × 2192 hourly portfolio snapshots each): level 7 → ~570 KB level 19 → ~489 KB (−14%) level 22 → ~489 KB (saturated) Decode speed is unchanged in zstd's design (decoder is independent of encoder level). For a 12,500-bundle workload (~64 GB at level 7) this trims the archive to ~55 GB at zero behavioural cost. Level 19 is the highest level still in the standard tier (no --ultra flag, no special memory window), so the bytes are readable by any stock zstd reader.
Captures the storage architecture proposal that came out of the
v8.9 size-review measurements:
- Why per-file compression has hit its ceiling (~14% headroom max)
- Three-tier model: Index (SQL) + Columnar bulk (Parquet) +
Content-addressed chunks (S3-compatible)
- .iafbt demoted from storage primitive to deterministic export
format; v1 and v2 remain readable forever
- Local-only OSS path (LocalTieredStore) keeps the framework
self-contained without a server
- Phased migration: v8.10 read-side helpers, v8.11 store
abstraction, Finterion remote store closed-source
Companion to docs/design/bundle-format-v2.md and
docs/design/ohlcv-dedup-protocol.md.
MDUYN
changed the title
…phase 1) Lift the existing untyped flat-row index helper into a public, typed Tier-1 contract: * New BacktestIndexRow dataclass (domain/backtesting/backtest_index_row.py) with identity / provenance / config / nested summary_metrics + forward-compat extras. Lossless to_flat_dict / from_flat_dict round trip for Parquet, SQL and JSON sinks. * New Backtest.index_row(bundle_path=None) method. Builds without decoding any v2 Parquet metric blobs, so it works against bundles loaded with Backtest.open(..., summary_only=True). This is the fast read path the upcoming 'iaf index' CLI (phase 2) and any tiered store implementation (phase 3) will rely on. * _backtest_to_index_row in backtest_utils now delegates to BacktestIndexRow.to_flat_dict() so the wire shape and the in-memory shape are a single source of truth (no behavioural change for the existing index.parquet sidecar). * Re-export BacktestIndexRow from the domain and top-level packages. * docs/design/tiered-backtest-storage.md \xa73.1 + roadmap row updated to reference the typed contract. Tests: 5 new (in-memory derivation, flat round-trip incl. NaN, unknown columns landing in extras, derivation from a summary_only=True bundle load). Full backtests suite green (29/29).
This was referenced May 10, 2026
CI flake8 flags the import as unused without the corresponding entry in __all__. Phase 1 missed adding it.
Tier-1 SQLite index over a folder of .iafbt bundles, building on the phase-1 BacktestIndexRow contract. * services/backtest_index/sqlite_index.py: SqliteBacktestIndex with create / open / upsert / upsert_many / iter_rows / query. Every scalar field of BacktestSummaryMetrics is promoted to its own summary_<name> SQL column so analysts can filter without opening any bundle (e.g. WHERE summary_sharpe_ratio > 1.0). parameters / strategy_ids / extras round-trip as JSON text. WAL mode for safe concurrent reads. Forward-only additive schema migration via PRAGMA user_version. * cli/index_command.py + new 'iaf index <bundle-dir>' click command: walks the directory, opens each bundle with summary_only=True (no Parquet metric-blob decode), derives BacktestIndexRow, upserts. --output, --absolute-paths and --no-progress flags. * docs/design/tiered-backtest-storage.md updated with phase-2 status. Tests: 12 new (8 SqliteBacktestIndex unit tests + 4 CLI integration tests via click.testing.CliRunner). Full repo suite green (1698 passed, 42 skipped). Lint clean.
feat(cli): `iaf index` + SqliteBacktestIndex — epic #540 phase 2
feat(backtest): BacktestIndexRow DTO + Backtest.index_row() — epic #540 phase 1
This was referenced May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundle format v2 + storage-layer groundwork
What's in this PR
1. Bundle format v2 (additive)
IAFBmagic + uint32 format version header for cheap routing/validation without decompressionBacktest.engine_type(vector/event/null), exposed asvector_runs/event_runsproperties; legacybacktest_runsstill works for engine-agnostic callersequity_curve,drawdown_series,cumulative_return_series,rolling_sharpe_ratio,monthly_returns,yearly_returns,twr_equity_curve,twr_drawdown_series) extracted as embedded Parquet blobs withint64epoch-ms timestamps +float64values2. Read-path additions (groundwork for the storage rewrite)
open_bundle(summary_only=True)andBacktest.open(summary_only=True)skip the per-run Parquet decode for bulk listing pipelines. Scalar summary metrics remain fully populated. This is the reader contract the futureBacktestStore.list()will sit on top of.save_bundle(format_version=, float32_ohlcv=)knobs for explicit downgrade and OHLCV size reduction.3. zstd default 7 → 19
4. Specs (the actual story)
docs/design/bundle-format-v2.md— public, stable wire format specdocs/design/ohlcv-dedup-protocol.md— content-addressed chunk negotiate protocol, generalizable to all chunk typesdocs/design/tiered-backtest-storage.md— the actual size story: relational index + columnar Parquet bulk + content-addressed chunks. Projects 64 GB → < 20 GB on the production-shape sample. This is what the storage rewrite epic will implement.Honest size accounting
On three real production-shape sample bundles in
docs/sample_backtests/(~570 KB each):The big win lives in the storage rewrite, not in the per-file format. This PR is the contract layer that makes the rewrite implementable without breaking anyone.
Compatibility
save_bundle()defaults to v2 — no caller change requiredformat_version=1to keep emitting legacy bundles for downstream tools that haven't upgraded→ Suggested release: minor (
v8.9.0).Tests
summary_onlymode, legacy default, round-trip preservation)What this PR does not claim
What it does do is freeze the v2 wire format, the
engine_typesemantics, and thesummary_onlyread path so that the storage rewrite can be implemented purely as additive changes on top.Follow-up
See the epic in #540 for the full tiered storage roadmap (replaces the now-closed #538 and #539).